A MULTILEVEL TEXTURE PROCESSING METHOD FOR MAPPING 
MULTIPLE IMAGES ONTO 3D MODELS 



BACKGROUND OF THE INVENTION 

Field of Invention 

5 The invention relates to a texture mapping process for 3D models for mapping images 

onto a 3D model. In particular, the invention is a multilevel texture processing method 
that processes images at the image level, the texture level, and then the pixel level and 
adjusts image pixel intensity on the texture. 

Related Art 

10 The texture mapping technology is used to increase the reality of computer generated 

images (CGIs) in the field of computer graphics. Through realistic images, the texture 
mapping technique provides better image rendering effects with fewer polygons of the 3D 
models. With the advance in integrated circuit (1C) designs, the operating functions and 
texture memory required for texture mapping have been incorporated into most of the 

15 graphic acceleration chips. Therefore, one can find many examples that use texture 
mapping to generate photo-realistic images in different kinds of applications, such as games, 
animations, and 3D websites. 

Generally speaking, to obtain a textured 3D model, one has to construct a complete and 
accurate digital 3D model of the object manually or by 3D scanning. The next step is to 

20 take 2D images of the object and to establish projection relations between the 3D model 
and the 2D images. After obtaining the 3D model of the object and the projection relation 
between it and some perspective image, we can project the digital 3D model onto a 2D 
image through the projection relation. In this manner, the image areas covered by the 
polygons in the 3D model are set to be the textures corresponding to the polygons. This 

25 completes the settings of the texture mapping. 
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When mapping the texture of an image to a 3D model, we can use the above-mentioned 
procedure to rapidly complete the settings of texture mapping, obtaining a textured model. 
However, an image is not sufficient to cover the texture of the whole surface of the object. 
In this case, the rendering of the model has a crack or a shading problem. Consequently, 
5 one needs images from different perspectives to completely determine the texture in each 
plaquette of the 3D model. When taking images of an object from different perspectives, 
there may be different rendering in colors for the same point on the object surface due to the 
facts that pictures are taken at different times, different light sources are used, and different 
camera settings (zooming, focus, camera position) are made. Such differences will result 
10 in variations in the texture of a digital 3D model, showing visual defects in the image 
rendering. 

To solve this problem, we can use a predetermined 3D model texture mapping to 
generate a global map through projections, such as projecting onto a cylindrical or spherical 
surface. The pixel color variations in each area are then adjusted manually or using 

15 existing image processing software such as PhotoImpact© and PhotoShop©. However, 
this work still requires a lot of time for a person familiar with image processing tools. For 
example, in the U.S. Pat. No. 6,057,850, a method has been proposed to use a lighting 
system that can be controlled and positioned to take images of an object with different 
lighting conditions. Finally, the images with fixed camera positions but different lighting 

20 conditions are stitched together. This is achieved by performing different levels of 
intelligent pixel-wise blending according to the lighting condition and the camera position. 
In this manner, shades due to different lighting conditions are eliminated by averaging 
according to their weights, providing an image with less dependence on the light sources. 
Afterwards, textures are cut from this image and put into the 3D model. This method 

25 allows one to rebuild a new image using different lighting conditions and camera positions. 
Nevertheless, it requires a special lighting system with the positioning function to obtain a 
better effect. 
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On the other hand, the U.S. Pat. No. 6,281,904 divides the 3D model into several 
different planar areas. The user manually selects a corresponding projection region in an 
image for each region on the 3D model. In different images, the projection regions for the 
same region on the 3D model are mixed together using various methods. This method 

5 concerns with how to synthesize the same region from different images but does not deal 
with differences among regions. To improve the situation, the U.S. Pat. No. 6,469,710 
proposes the concept of alpha blending. The contribution of each pixel in the projection 
regions of different images for the same 3D model surface region to the final texture is 
determined. If there are images of other objects in the projection region, the contribution 

10 of this pixel is set to be zero, eliminating effects of erroneous pixels. However, the 
operation is the projection of the same 3D model surface region. The texture variation 
among regions will still have different visual effects. 

SUMMARY OF THE INVENTION 

In view of the foregoing, the invention provides a multilevel texture processing method 
15 for mapping multiple images onto 3D models. Using this method, the textures of 
neighboring plaquettes do not have discontinuous variations in the pixel intensity due to 
different lighting conditions and camera parameter settings. This preserves the 
photo-realistic quality of the computer-generated image (CGI). 

The disclosed method mainly divides the image mapping process into three levels: the 
20 image level, the texture level, and the pixel level. The pixel intensity in each level is 
adjusted at different strengths. At the image level, the pixel intensity of overlapped 
plaquettes is computed by weighted average, and the whole image is adjusted accordingly. 
The adjustments at the texture level use predetermined conditions to take one of the 
textures in the overlapped plaquettes or a normalization operation to compute the required 
25 texture. It further blurs the textures of neighboring plaquettes so that it becomes smoother. 
Finally, the pixel-level step processes the colors of pixels inside the plaquettes to obtain an 
optimized 3D model. At the same time, the invention can perform accumulated processes 



3 



on input image data, so that no new calculations are needed when new images are entered. 
This can save much time in processing. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The invention will become more fully understood from the detailed description given 
5 hereinbelow illustration only, and thus are not limitative of the present invention, and 
wherein: 

FIG. 1 is a schematic flowchart of the invention; 

FIG. 2 is a flowchart for processing at the image level; 

FIG. 3 is a flowchart for processing at the texture level; 

10 FIG. 4 is a schematic view of the neighboring plaquettes; and 

FIG. 5 is a flowchart for processing at the pixel level. 

DETAILED DESCRIPTION OF THE INVENTION 

With reference to FIG. 1, the disclosed multilevel texture processing method for 
mapping multiple images onto a 3D model starts by providing several images to a 3D 

15 model (step 101). The sources of the images can be obtained by taking pictures of an 
object from different perspectives. It does not need any special lighting system as in the 
prior art. Then the method determines whether the 3D model is a textured model (step 
102). If the input image is the first image to be pasted on the 3D model, then the answer is 
obviously "no," and the texture extraction and mapping start (step 103). If the input image 

20 is not the first image, then the answer is "yes," and the image is adjusted in order at the 
image level (step 104), the texture level (step 105), and the pixel level (step 106). 
Afterwards, the method determines whether there is a following image (step 107). After 
all images are processed, the method outputs the 3D model (step 108). An advantage of 
this method is that no new calculation has to be made. Accumulated calculations can be 
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made and accumulated calculations can be done at any time a new image is added. 

As said before, the texture extraction and mapping start when the input image is the 
first one to be mapped onto the 3D model (step 103). After an image and a 3D model are 
loaded into memory, the projection matrix of the image is first determined manually or 
5 from an automatic camera calibration procedure. Afterwards, the polygons of the digital 
3D model are projected onto the real images using the projection matrix. If a polygon 
projected on the image is visible, then the region covered by the polygon is set to be its 
texture; otherwise, it is skipped and the procedure continues to the next polygon. The 
procedure is executed iteratively until all the polygons are processed. At this moment, 
10 since there is only a single image, there may be many blank polygons without textures 
attached on the 3D model. 

After the 3D model is textured with the. image, if there are other images to be extracted 
and mapped onto the model, images they are processed with the following three-level 
procedure, the image-level adjustments (step 104), the texture-level adjustments (step 105), 

15 and the pixel-level adjustments (step 106). Please refer to FIG. 2. Before two images are 
combined, the method first divides the 3D textured model and the image into projections of 
several polygons (step 201). Of course, the same spatial coordinates have to be used 
during the division transformation. The overlapped polygons are then extracted (step 202). 
These overlapped polygons are used to compute the statistics of the brightness of the pixels 

20 (step 203). The pixel intensity of the whole image is then adjusted according to (step 204). 
The formula as follows: 

where ju s is the averaged pixel intensity of the overlapped polygon on the 3D model; ju b 
is the averaged pixel intensity of the overlapped polygon of the input image; I s ( x„yj ) is the 
25 pixel intensity of each point on the 3D model: and I s ( x 1? y, ) is the adjusted pixel intensity 
of each point on the 3D model. 
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In other words, the method uses the statistics distribution of pixel intensity of the 
overlapped polygons to adjust the pixel intensity of the whole image. Therefore, the pixel 
intensity receives a preliminary adjustment immediately after the image is entered. 

After the pixel intensity of the image is adjusted, the method continues to reconfigure 
5 the texture map settings in the overlapped polygons (step 205). The adjustment is made 
according to the resolution, polygon orientation, viewing perspective, and so on. For 
example, if one uses the resolution as the determination criterion, then the polygon texture 
with a larger resolution can be directly used as the final polygon texture. The other 
conditions can be similarly applied. 

10 After the image-level adjustments (step 104) and texture map reconfiguration (step 

205), the method performs the texture-level adjustments to smooth the texture of the 
polygons. As shown in FIG. 3, the smoothing can be achieved using texture normalization 
(step 301) and texture blurring (step 302). The texture normalization (step 301) mainly 
takes the averaged pixel intensity of the texture in the overlapped polygon and feeds into 

1 5 the following formula: 



where ju s is the averaged pixel intensity of the overlapped polygon on the 3D model; fi b 
is the averaged pixel intensity of the overlapped polygon on the input image; T s (xj,yj ) is 
the texture pixel intensity at each point in the polygon; and T s ( Xj,yj ) is the adjusted texture 
20 pixel intensity at each point in the polygon. 

The method then performs texture blurring (step 302). The averaged intensity fi s o of 
the texture corresponding to each polygon is first computed (FIG. 4). The averaged 
intensities ju s o ~ tt sio of the surrounding polygon textures are also computed. The 
formula 



7 i , (w/) = 7 ',(w#)-/', +V h 
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is then used to compute the destination intensity. The weight is determined using different 
conditions (such as distance and brightness). This makes the texture of each polygon 
match with its surrounding ones. The boundaries of neighboring plaquettes are thus 
blurred and smoothed. Of course, the polygons shown in the drawing are triangular. 
5 However, any geometrical shape can be used. 

Finally, step 106 performs the pixel-level adjustments. With reference to FIG. 5, an 
arbitrary polygon is selected (step 501). The method determines whether it has a 
discontinuous color variation from the neighboring polygon textures (step 502). If there is 
a discontinuous color change between the textures, then pixel intensity modification process 

10 is invoked. Otherwise, the method updates internal texture pixels (step 506) and ensures 
all the other polygons if there are any not processed yet (step 507). The pixel intensity 
modification process consists of threes steps: Firstly, a pixel, p, is extracted from the 
boundary (step 503). The method searches another image point in neighboring plaquette 
textures that is closest to the boundary pixel p. The pixel intensity of the boundary image 

15 point is set to be weighted average of the two (step 504). The pixel intensities before and 
after the adjustment are recorded so that the difference can be used to adjust the intensities 
of other pixels inside the polygon (step 505). The pixel intensity adjustment inside the 
polygon is done using the following formula: 

20 where w, is a relevant weight; Idj is the adjusted difference; N is the number of total 
adjustments; T s ( x„yi ) is the pixel intensity of each point on the 3D model; and T s : ( x l5 y, ) 
is the adjusted pixel intensity of each point on the 3D model. 

In the above formula, we may use the pixel intensity difference of a single (N=l) 
nearest boundary pixel or pixels on different boundaries (N = the number of sides in a 
25 polyhedron). This helps achieving smooth visual effects. The color and brightness 
variations between two regions can be eliminated. 
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Certain variations would be apparent to those skilled in the art, which variations are 
considered within the spirit and scope of the claimed invention. 
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